1. Field of the Invention
The present invention relates generally to Java testing, and more particularly to automated Java specification tracking in a Java compatibility-testing environment.
2. Description of the Related Art
Currently, Java environments can be categorized into various Java technologies. A Java technology is defined as a Java specification and its reference implementation. Examples of Java technologies are Java 2 Standard Edition (J2SE), Java 2 Enterprise Edition (J2EE), and Mobile Information Device Profile (MIDP). As with most other types of Java software, a new Java technology should be tested to assure consistency across multiple platforms. This testing is generally performed using compatibility testing.
Compatibility testing refers to the methods used to test an implementation of a Java technology specification in order to assure consistency across multiple hardware platforms, operating systems, and other implementations of the same Java technology specification. When this assurance is accomplished by means of a formal process, application developers can then be confident that an application will run in a consistent manner across all tested implementations of the same Java technology specification. This consistent specification-based behavior is a primary function of compatibility testing.
Compatibility testing differs from traditional product testing in a number of ways. Unlike product testing, compatibility testing is not primarily concerned with robustness, performance, or ease of use. The primary purpose of Java compatibility testing is to determine whether an implementation of a technology is compliant with the specification of that technology.
Compatibility test development for a given feature relies on a complete specification and reference implementation for that feature. Compatibility testing is a means of ensuring correctness, completeness, and consistency across all implementations of a technology specification that are developed. The primary goal of compatibility testing is to provide the assurance that an application will run in a consistent manner across all tested implementations of a technology.
To determine if the implementation of a particular Java technology is compliant with the specification for the particular Java technology, technology compatibility kits (TCK) may be used. A TCK is a suite of tests, tools, and documentation that allows an implementor of a Java technology specification to determine if the implementation is compliant with the specification.
A TCK typically includes a Test Harness, defined as the applications and tools that are used for test execution and test suite management, and a TCK Test Suite, which is the composite of the actual test cases in a TCK that are executed to test an implementation. A TCK can also include documentation that includes the specific TCK usage procedures, and the compatibility testing requirements that apply to the related technology release (usually in the form of a TCK user's guide). Also, a description of the TCK appeals process can be included, as well as an audit process, which is used to better ensure the integrity of a consistent self-testing compatibility program.
As mentioned above, a TCK usually includes a TCK test suite, which is a set of tests designed to verify that an implementation of a Java technology complies with the appropriate specification. Each test in a TCK test suite is composed of one or more test cases that are designated by a test description. A test case is the source code and accompanying information designed to exercise one aspect of a specified assertion. Accompanying information may include test documentation, auxiliary data files and other resources used by the source code.
In order to be complete, a test suite includes a test case to verify each and every testable assertion that is made by the API specification. Test developers must review the actual specification document and generate at least one test case for each testable assertion that appears in the API specification.
Unfortunately, the conventional method for determining assertions for a particular specification is a laborious process involving a manual inspection of the specification. Prior to the test design stage during conventional testing, the test developer must scan through the specification and split the entire text into logical statements. Each logical statement then needs to be examined by type to indicate if it is a testable assertion.
Of course, the process of manually inspecting the specification is a time consuming process prone to errors. Moreover, since one of the most important issues of the TCK development process is to keep TCK tests synchronized with their technology's current API specification, the manual inspection process is exacerbated whenever the specification changes. Upon any specification change, the corresponding TCK tests must be revised and corrected if their functionality has been affected by the change. To accomplish this, the specification must be re-inspected to confirm the current assertions and determine if particular assertions should be removed or additional assertions should be added.
In view of the foregoing, there is a need for methods for tracking the specification to determine assertions. Preferably, the methods should be automated, and should provide tracking between different versions of a specification to identify TCK test that are affected by each change assertion.